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This  report  describes  the  design  and  evaluation  of  classifiers  for  distinguishing 
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and  Recognition  System  (OLPARS).  The  data  base,  supplied  by  RADC  (COTI),  consisted 
of  a  set  of  measurements  extracted  from  seismometer  responses  to  each  of  the  intruding 
targets. 
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ABSTRACT 


This  report  describes  the  design  and  evaluation  of  classi¬ 
fiers  for  distinguishing  humans*  vehicles,  aircraft  and  background 
alarms  based  on  seismic  disturbances.  The  design  and  evaluation 
was  performed  by  RADC  (ISCP)  using  the  On-Line  Pattern  Analysis 
and  Recognition  System  (OLPARS).  The  data  base,  supplied  by 
RADC  (COTI),  consisted  of  a  set  of  measurements  extracted  from 
seismometer  responses  to  each  of  the  intruding  targets. 
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SECTION  I 


INTRODUCTION 

The  report  discusses  the  design  and  evaluation  of  two  classi¬ 
fiers  to  distinguish  human  intrusions,  vehicular  intrusions, 
aerial  intrusions,  and/or  background  alarms  based  on  seismic 
disturbances.  The  design  is  based  on  data  supplied  by  RADC  (COTI). 
The  data  consisted  of  a  set  of  measurements  extracted  from  the 
outputs  of  vertical  axis  seismometers.  Consequently,  the  design 
by  RADC  (ISCP)  was  restricted  to  starting  with  a  preselected  set 
of  segmentation  criteria  and  features  already  incorporated  into 
the  data  base. 

The  main  results  consist  of: 

.  An  increase  in  the  probability  of  detection  (on  the 
design  set)  from  83. 6X  to  95%  over  the  previously 
supplied  three-class  design. 

.  A  reduction  in  design  set  error  rate  of  33%  over  the 
previously  supplied  COTI  three-class  design  (humans, 
vehicles,  and  nuisances).  The  error  rate  is  defined  as 
the  percentage  of  samples  mi s c 1  as s i f i ed .  Although  a 
further  reduction  in  the  design  set  error  rate  is  possible, 
no  additional  improvement  in  field  performance  could  be 
expected . 

.  Independent  testing  which  gives  a  much  better  estimate 
of  field  performance. 

.  A  new  classifier  which  identifies  aerial  intrusions  as 
a  separate  class. 
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.  One  design  which  may  be  used  as  a  ,  <-  ,  or  h  —  o  1  a:- .■> 

classifier  with  the  simple  add i t ion /de let  ion  of  QR-gate 
connections. 

SECTION  II 

THE  DATA  BASE 

The  data  base  supplied  by  COTI  consists  of  a  set  of  seven¬ 
teen  (IT)  measurements  extracted  from  the  raw  seismic  datu. 

The  measurements  consisted  of  eight  Ti  n.e-Bet  ween  -  Even  t -H  i  s  t  opr  nr; 
Cells  (TBE),  eight  Ti me-Between- Zero  Crossing  Histogram  Celle 
( ZC )  ,  and  the  time  period  for  which  a  preliminary  segmentation 
algorithm  indicated  that  an  event  of  interest  was  preser.t. 

Table  II-l  gives  further  details  on  these  measurements. 

The  classes  consisted  of  (1)  single  humans  walking  or  run¬ 
ning  at  various  ranges,  {?)  wheeled  vehicles  at  various  speeds, 
ranges,  and  weights,  (3)  false  outputs  cf  the  segmentation 
routine  caused  by  environmental  disturbances,  such  as  noise, 
wind,  rain,  or  lightning,  and  (  i* )  helicopters  and  aircraft  (both 
prop  and  Jet)  at  various  altitudes  and  speeds. 


TABLE  11-1  FEATURES  TABLE 


TIME-BETWEEN-EVENT  CELLS  (MS)  COTI-DESIQN  ISCP-DESIGN 

FEATURES  FEATURES 


1 

200-280 

1 

1  /IT 

2 

280-360 

2 

2/  IT 

3 

360-1*1*0 

3+U 

3/17 

k 

1*1*0-520 

WIT 

5 

520-600 

5+6 

5/  IT 

6 

600-680 

6/17 

i 

680-T60 

7  +  8 

T  /IT 

8 

>  760 

8/17 

ZERO-CROSSING 

CELLS  (MS) 

9 

0-12 

9 

9/17 

10 

12-21* 

10 

10/1  r 

1  1 

2 1*-  36 

11/17 

12 

36-1*8 

12/17 

13 

U8-60 

13/17 

lh 

60-72 

13+lU+15*l6 

1 W 17 

15 

72-8U 

15/17 

in 

>  6U 

16/17 

TIME  ON 

IT  IT 


•Division  by  feature  IT*  normalizes  the  feature  on  a  per  time 
basis  . 
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SECTION  III 


TI1HEE  CLASS  CLASSIFIERS 

A.  COT1  -  SUPPLIED  DESIGN 

In  the  three-class  problem,  the  background  and  aerial 
intrusions  are  treated  as  a  single  class  labeled  nuisances  ( tl  )  . 
Using  the  CQTI-Supplied  design  logic,  the  data  supplied  by  COTI 
was  checked  and  a  confusion  matrix  was  prepared  on  a  run,  us 
opposed  to  an  intrusion,  basis.  Some  clarification  of  intru¬ 
sion  and  run  basis  is  needed.  Due  t.c  the  COT  T -Sup  p  1  i  o  d  segmen¬ 
tation  routines,  it  is  possible  for  either  a  single  human  or  n 
single  vehicular  intrusion  to  cause  the  elacsifior  to  trigger 
and  output  a  string  or  sequence  of  decisions.  A  single  output 
is  called  a  run ■  Although  results  discussed  elsewhere  are  cn  a 
run  basis,  this  report  emphasizes  results  on  an  intrusion  basis 
since  in  the  field  there  exists  no  method  for  defining  an  intru¬ 
sion  as  a  unique  entity.  However,  there  is  little  difference 
between  either  method  of  tabulating  results. 

Table  III-I  shows  the  confusion  matrix  and  associated 
classification  and  error  rates  for  the  COTI  supplied  design. 

It  is  emphasized  that  these  rates  are  the  design  set  rates 
only,  and  that  design  set  rates  tend  to  be  optimistic  (see 
Foley  ( 1 ! ) . 
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TABLE  III-l  COTI  THREE  CLASS  DESIGN  SET  RESULTS 

(RUN  BASIS) 


TRUE 

CLASS 


ASSIGNED  CLASS 


H 

V 

N 

Human  (H) 

301 

16 

85 

Vehicle  (V) 

12 

168 

13 

Nuisance  ( n ) 

6 

51* 

1059 

NO.  OF  RUNS 
1*02 
193 
1119 


^Correct  Classifi¬ 
cation 


Detection 


=  Prob  {H  is  H  or  V  is  V  or  N  is  N}  - 


=  99.1 5 

171L 


,  >  1*97 

Prob  iH  or  V  is  called  H  or  VI  = 


=  83.655 


^False  Alarm 


Prob  {N  is  V  or  H}  *  =  5*^ 

1119 


B.  ISCP  THREE-CLASS  DESIGN 


The  design  by  ISCP  was  restricted  by  the  data  base  supplied 
by  COTI  in  the  following: 

1.  Only  the  classes  on  which  data  was  collected  could  be 
identified. 

2.  ISCP  was  restricted  to  using  the  basic  features  or 
measureiaei.es  extracted. 

It  became  obvious  that  each  of  the  first  sixteen  measurements 
which  were  collected  over  a  period  cf  time  which  varied  from  3  to 
32  seconds,  needed  come  form  of  time  normalization  to  reduce  the 
wide  within-class  variance  cf  each  measuremen  Consequently, 
the  first  sixteen  features  were  normalized  by  viding  by  the 
time-on  measurement  (#  17)-  Therefore,  the  zero  crossing  and 
time-between-event  measurements  are  now  on  a  unit  time  basis. 
Based  on  this  normalized  set  of  measurements,  the  On-Line  Pattern 
Analysis  and  Recognition  System  (OLPARS),  (reference  Sammor.  (2)), 
was  used  to  design  entirely  new  classification  logic.  This  logic 
consists  solely  of  sets  of  linear  discriminants  for  ease  of  hard¬ 
ware  implementation.  The  logic  is  based  on  the  pairwise  Fisher 
Linear  Discriminant  Technique. 

For  each  pair  of  classes  i  and  J,  a  unit  vector  is 

computed  such  that  projections  of  the  data  d^j  maximizes  the 
ratio  of  the  be tve e n -c las s  scatter  to  the  within-class  scatter. 
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It  has  been  shown  that  the  direction  d^j  which  maximises  this 
ratio  is  given  by  [2] 

?ij  =  a  WiJ  *ij 

where  Wij  =  (Ni  -  1)  Ci  ♦  ( N j  -  1)  Cj 

*  Estimated  covariance  matrix  for  class  i 

^i  J  =  Hi  '  H J 

V ^  ~  Estimated  mean  vector  of  class  i 

and  a  is  a  normalizing  constant  so  that  |d|  =  1 


OLPARS  computes  d ^ j  for  all  possible  pairs  of  classes  and 
establishes  an  initial  threshold. 


The  classification  rule  for  the  pairs  of  classes  i,  J  is  given  by 

XT  djj  >  9ij==>  X  is  a  member  of  class  i 
where  X  is  the  ordered  set  of  measurements 

The  0^j  computed  by  OLPARS  may  be  adjusted  by  the  user  to  obtain 
optimal  discrimination  along  each  d^j.  A  frequency  histogram  of 
the  sample  vectors  projected  on  the  appropriate  discriminants  can 
be  displayed  for  each  pair  of  classes.  The  histogram  is  obtained 
by  summing  the  number  of  sample  vectors  that  fall  in  each  of 
120  bins  that  spar,  the  range  of  the  projections.  For  viewing 
purposes,  the  frequency  bins  are  displayed  as  to  or  less  columns 
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of  characters  on  the  CRT.  The  0^  computed  by  the  system  is 

displayed  as  a  vertical  line  on  the  CRT.  The  user  can  adjust 

the  Q.  by  designating  a  display  column  on  the  CRT  or  more 
t  j 

accurately  by  inputing  a  specific  bin  number.  A  hardcopy  of 
the  projected  data  values  versus  bin  number  is  provided  by  the 
system  to  assist  the  user  in  selecting  an  optimal  threshold. 

The  discriminant  vectors,  weights  and  thresholds  for  each 
pair  of  classes  are  given  in  Appendix  A.  The  input  consists  of 
the  normalized  set  of  17  features  given  in  Table  II-l.  bach 
input  vector  is  classified  using  the  scheme  diagramed  in  Figure 
III-l.  Although  the  logic  scheme  is  shown  for  three  classes,  it 
may  be  easily  extended  to  an  arbitrary  number  of  classes.  Table 
1 1 1 -  2  shows  the  results  of  the  ISCP  designed  logic. 


C.  COMPARISON  AND  ERROR  ANALYSIS 

The  main  improvements  are  an  increase  in  the  probability  of 
detection  from  83.6?  to  95?  and  a  33?  reduction  in  the  Error  Rate 
from  10.9?  "o  7.3%.  The  relatively  small  increase  in  the  prob¬ 
ability  of  false  alarm  is  not  considered  significant  when  compared 
to  the  increase  in  the  probability  of  detection.  The  decision  was 
made  to  allow  9  more  false  alarms  so  that  68  more  detections  could 
be  realized.  These  are  design  set  rates.  As  pointed  out  [l], 
these  predictors  of  performance  may  be  optimistic  estimates  of 
field  performance.  However,  since  this  is  the  only  estimate  used 
in  the  previously  supplied  design  ,  these  estimates  are  the  only 
available  means  of  comparison. 

It  is  possible  for  ISCP  to  "over-design"  on  the  data  base  in 
order  to  lower  the  design  set  error  rate.  however,  those  changes 
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PAIRWISE  FISHER  LOGIC 


INNER 

PRODUCT  COMPARATORS  COUNTERS 


H: 

Human 

V: 

Vehicle 

N: 

Nuisance  (i.e..  Background 

or  Aircraft) 

X: 

Unknown  Input  (i.e.  output 
and  feature 

of  the  segmentation 
extractor  device) 

dh 

Linear  Discriminant 

0: 

Threshold 

: 

Inverter 

FIGURE  III-l 


*  I  f  vote  is  tied,  assign  input  to  class  with  the  highest  u  priori 
probability. 


MAXIMUM  SELECTOR 
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could  not  be  expected  to  improve  the  field  performance  of  the 
device.  The  errors  are  present  due  to  the  somewhat  unpredictable 
multiple  turn-on/turn-off  criteria  used.  For  instance,  the 
t ime-between-events  histogram  feature  was  chosen  in  order  to 
discriminate  the  class  of  humans  from  the  remaining  classes. 

The  distinguishing  characteristics  for  the  human  class  were 
hypothesized  to  be  significant  entries  in  the  higher  order 
cells  due  to  the  sequence  of  footsteps.  Consider  human  intrusion 


numb  e  r 

626,  run  numbers 

one  and  five  as  given 

in  Table 

Iz. 

B-E  Cell  Numbers 

RUN  #  1 

RUN  #  5 

1. 

200-280  MS 

159 

5 1< 

2. 

280-360 

0 

1 

3. 

36O-I4L0 

0 

3 

L  . 

1*1*0-520 

0 

1 

5- 

520-600 

0 

7 

6 . 

600-680 

0 

19 

7. 

68O-76O 

0 

3 

8. 

>  760 

0 

1 

TIME-BETWEEN-EVENT  FEATURES  FOR  HUMAN  INTRUSION  626 

TABLE  II  I- 3 


•Since  only  runs  1  and  5  were  supplied,  it  is  assumed  that  the 
missing  run  numbers  are  due  to  non-seismic  disturbances. 
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ISCP  THREE-CLASS  DESIGN  SET  RESULTS 


(RUN  BASIS) 


ASSIGNED  CLASS 


H 

V 

N 

NO.  OF  RUNS 

Human  ( H ) 

363 

17 

22 

1*02 

TRUE 

Vehicle  (V) 

9 

176 

8 

193 

CLASS 

Nuisance  (N) 

23 

1*6 

1050 

1119 

1711* 

pCorrect  Classif ication~Prob 

(H  is  H 

or 

Vis  V 

or  N 

is  N }  = 

I589 

171** 

=  92 .1% 

^Detection  B  Prob 

{ H  or 

V  is 

H  or 

V}  = 

=  95% 

595 

PFalse  Alarm  =  Prob 

(N  is 

H  or 

V}  = 

69 

=  6.2% 

1119 

TABLE  1 1 1 - 2 
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could  not  be  expected  to  improve  the  field  performance  of  the 
device.  The  errors  are  present  due  to  the  somewhat  unpredictable 
multiple  turn-on/turn-off  criteria  used.  For  instance,  the 
t ime-between-e vents  histogram  feature  was  chosen  in  order  to 
discriminate  the  class  of  humans  from  the  remaining  classes. 

The  distinguishing  characteristics  for  the  human  class  were 
hypothesized  to  be  significant  entries  in  the  higher  order 
cells  due  to  the  sequence  of  footsteps.  Consider  human  intrusion 
number  626,  run  numbers  one  and  five  as  given  in  Table  III-3. 


T-B- 

E  Cell  Numbers 

RUN  ft  1 

RUN  ft 

1. 

200-280  MS 

159 

5U 

2. 

280-360 

0 

1 

3. 

360-LL0 

0 

3 

U. 

LLO-520 

0 

1 

5  • 

520-600 

0 

7 

6 . 

600-680 

0 

19 

Y. 

680-760 

0 

3 

8. 

>.  760 

0 

1 

TIME-BETWEEN-EVENT  FEATURES  FOR  HUMAN  INTRUSION  626 

TABLE  1 1 1-3 


•Since  only  runs  1  and  5  were  supplied,  it  is  assumed  that  the 
missing  run  numbers  are  due  to  non-seismic  disturbances. 
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First  notice  the  large  differences  in  the  numerical  values 


of  features  for  two  runs  from  the  seme  intrusion.  Second, 
notice  that  while  footsteps  are  evident  in  run  5,  none  are 
evident  in  run  1. 

As  &  second  example,  consider  vehicular  intrusion  number 
h02,  runs  U  and  7  (again  these  are  the  only  seismic  runs 
supplied)  given  in  Table  Ill-t.  Hun  t  contains  little,  if  any, 
indication  of  footsteps  and  is  assigned  by  the  OLPARS  logic  to 
the  class  of  vehicles.  However,  run  9  indicates  footsteps, 
and  (we  submit)  is  "correctly"  classified  as  a  man.  The 
point  is: 

The  logic  is  performing  exactly  as  it  should,  if  it 
assigns  spurious  outputs  of  the  segmentation  logic 
to  the  classes  these  spurious  outputs  most  closely 
resemble.  In  other  words,  these  errors  are  inherent 
in  the  data  base,  and  we  submit  that  these  errors 
are  due  to  unpredictable  outputs  of  the  multiple 
turn-on/turn-of f  segmentation  criteria. 


TIME- BETWEEN- EVENTS  FEATUFES  FOR  VEHICULAR  INTRUSION  U02 


TABLE  I I I - L 


T-B- 

E  Cell  Numbers 

RUN  H  It 

RUN  1 

1 . 

200-280 

39 

2 

2. 

280-360 

2 

0 

3- 

360-U0 

O 

c. 

1 

It  . 

IiUO-520 

0 

3 

5- 

520-600 

0 

17 

6. 

600-680 

0 

19 

7  . 

680-760 

0 

6 

8. 

>  760 

2 

3 

Appendix  B  gives  the  usefulness  of  each  feature  for  dis¬ 
criminating  each  pair  of  classes.  'l’ne  ranking  procedure 
measures  the  overlap  l'cr  each  pair  of  classes  a]  ong  each  feature 
This  procedure  called  the  Probability  of  Confusion  Measure  is  a 
nistogram  approximation  of  the  class  distributions  along  each 
feature.  The  technique  is  particularly  useful  for  data  sets  in 
which  the  modality  is  unknown  since  it  is  inaepenuent  of  mean 
and  variance  statistics. 

The  procedure  yields  three  measures  for  each  feature: 

(1)  a  pairwise  measure  for  differentiating  class  i  from  class  j, 

(2)  a  measure  for  differentiating  class  i  from  all  other  classes 
and  (  3 )  a  measure  of  significance  for  each  feature  for  differen¬ 
tiating  each  pair  of  classes  [  !i  j  . 


Those  feature  ratings  demor.  s  t  r  ate  an  interest  ini'  point. 
Although  it  has  been  hypothesized  that  the  higher  order  time- 
between-event  cells  were  useful  in  discriminating  humans, 
Probability  of  Confusion  Measure  rates  these  as  the  poorest 
features  for  this  purpose. 

D.  INDEPENDENT  TESTING  OF  ISCP  DESIGN 

Although  a  large  number  of  samples  were  supplied  for  each 
class,  it  is  always  prudent  to  use  independent  (different) 
data  for  the  design  ana  testing  of  the  classifier.  It  is 
also  important  to  note  that  it  is  possible  to  test  the  logic 
without  having  to  implement  tne  device.  Independent  testing 
has  been  Shown  to  te  a  -  r,  better  estimate  (i.e.  an  unbiased 
estimate)  of  the  field  performance  of  the  design  (e.g.  Foley 
[1]). 

Although  no  independent  testing  of  the  supplied  design 
was  conducted,  the  following  results  on  the  ISCP  design  are 
available.  The  design  and  test  sets  each  consisted  of  approxi¬ 
mately  of  the  data  base  selected  on  a  random  basis.  Table 

III-5  shows  the  design  and  tesu  set  results. 

The  Tables  III-,?  and  1 1 1  -  5  can  be  interrupted  in  the 
following  manner.  The  expected  field  results  should  be  greater 
than  the  test  results  on  naif  the  data  (Table  1 1 1  —  5 )  •  but  less 
than  the  design  results  on  all  the  data  (Table  III-2).  For 
example,  the  probability  of  detection  should  lie  between  91. 
and  95*0/1. 
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DESIGN  AND  TEST  HESULTS 
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13  19 
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17 

32  511 

560 

852 

862 

Pcc 

799 

"  852  = 

93.8* 

Pcc 

771 
■  862 

*  89-3* 

Pd 

_  281.  _ 
293 

95-9* 

pd 

.  ill 
302 

=  91-7* 

PFA 

559 

5-7* 

PFA 

*  **9 

5 6Z 

*  8.7* 

TABLE  I I I- 5 
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SECTION  IV 


FOUR-CLASS  CLASSIFIER 

For  this  classifier,  the  classes  of  interest  are  humans, 
vehicles,  aerial  intrusions,  and  background  disturbances.  Tne 
data  for  aerial  intrusions  include  helicopters,  and  both  prop 
and  Jet  aircraft. 

Usually  when  a  classifier  is  designed,  features  (measurements) 
are  chosen  which  hopefully  contain  discriminatory  information  for 
each  class.  Since  the  features  were  previously  selected  for  only 
three  classes,  no  features  could  be  selected  with  the  intention 
of  identifying  aircraft .  Despite  this  handicap,  a  four-class 
classifier  was  designed  and  the  results  are  given  in  Table  Ill-b 
for  the  design  set  results  on  the  entire  data.  Table  1 1 1  —  7 
contains  the  results  where  half  the  data  was  used  for  design  r.na 
half  for  test.  Notice  that  the  definition  of  detection,  classi¬ 
fication,  and  false  alarm  c h  ange ,  since  a  new  class  has  been 
added. 

The  majority  of  errors,  approximately  9i ,  are  due  to  the 
class  of  aerial  intrusions.  In  fact,  these  results  are  quite 
encouraging  since  another  study  indicated  that  the  addition  of 
new  classes  seriously  effect  the  recognition  accuracies.  The 
addition  of  features  for  distinguishing  between  pron  aircraft, 

Jet  aircraft,  and  helicopters  offers  the  possiblity  of  even 
better  classifiers. 
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D E SIGN  SET  RATES  ON  ENTIRE  DATA  SE T 


ASSIGNED  CLASS 


H 

V 

B 

A 

NO.  OF  HUNS 

Humans 

H 
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17 

10 

12 

hO  2 

True 

Vehicle  s 

V 

9 

176 

0 

8 
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Class 

Background 

B 

10 

h 

339 

67 

h20 

Aerial  I n- 
trus ion 

A 

13 

h2 

81 

563 

699 

17U 

Pc  c 

=  Prob  {H 

i  s 

H  or  A  is  A  or  B 

is  BorV  is  V }  * 

lUf  -  8)4 
17 1  ^ 

PD 

*  Prob { ( H 

or 

V  or  A)  }  « 

129h 

=  92.9* 

PFA 

*  Prob  {B 

is 

( H  or  V  or  A) }  = 

iti  *  19-3‘ 

TABLE  III-6 


DESIGN  AND  TEST  SET  RESULTS 


DESIGN 

TEST 

11 

V  B 

A 

§  OF  RUNS 

H 

V 

B 

A 

*  OF  RUNS 
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7  1 

5 

199 

H 
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6 

8 

6 

203 

V 

2 

89  0 

3 

91 

V 

12 

76 

1 

10 

99 

D 

6 

1  173 

30 

210 

B 

7 

3 

1 61 

36 

210 

A 

7 

18  39 

ro 

00 

319 

A 

10 

29 

3  6 

27  3 

350 

352 

862 

^0 

o 

o 

=  m 
852 

=  65-5? 

Pec 

=  6£6 
862 

=  80 

.  7% 

Pa 

=  ii8 

=  Q3.lt 

pd 

=  £0* 

•  92 

■  7  « 

612 

652 

PFA 

=  _il 
210 

*  17.6* 

PFA 

_  16 
210 

=  21. 

.  9f. 

TABLE  1 1 1-7 
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SECTION  V 


DISCUSSION  AND  RECOMMENDATIONS 

This  report  demonstrates  the  capability  of  ISCP  to  design 
superior  classification  logic,  using  the  OLPARS  System,  in  short 
periods  of  time  given  a  vector  data  base.  Of  course,  the  quality 
of  the  classifier  cannot  exceed  the  quality  and  universality  of 
the  input  data  base.  It  appears  that  further  improvements  in 
classifier  designs  will  require  new  and  improved  segmentation 
procedures  and  additional  and  more  relevant  features. 

However,  one  possible  extension  using  the  present  data  base 
may  yield  a  further  improvement.  For  a  single  intrusion  the 
segiuent&t  ion  procedure  may  yield  multiple  outputs  or  runs.  In 
general,  the  runs  corresponding  to  strong  signals  are  classified 
correctly.  However,  due  to  the  somewhat  unpredictable  nature 
of  the  segmentation  procedure,  weak  runs  can  result  and  are 
incorrectly  classified.  In  the  present  system,  each  run  is 
given  equal  weight.  However,  it  is  possible  to  create  regions 
of  either  rejection  or  low  confidence.  If  this  strategy  were 
implemented,  the  user  of  the  system  could  be  supplied  not  only 
with  a  sequence  of  decisions,  but  also  informatior  indicating 
the  level  of  confidence  in  each  decision. 
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PROBABILITY  OF  CONFUSION  MEASURES 
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